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ABSTRACT 

Objectives: To undertake an assessment of the 
association between coproduction and satisfaction with 
decisions made for local healthcare communities. 
Design: A coproduction scale was developed and 
tested to measure individual National Health Service 
(NHS) commissioners' satisfaction with commissioning 
decisions. 

Setting: 11 English Primary Care Trusts in 2010-2011. 
Participants: Staff employed at NHS band 7 or above 
involved in commissioning decisions in the NHS. 345/ 
440 (78%) of participants completed part of all of the 
survey. 

Main outcome measure: Reliability and validity of a 
coproduction scale were assessed using a correlation- 
based principal component analysis model with direct 
oblimin rotation. Multilevel modelling was used to 
predict decision satisfaction. 
Results: The analysis revealed that coproduction 
consisted of three principal components: productive 
discussion, information and dealing with uncertainty. 
Higher decision satisfaction was associated with smaller 
decisions, more productive discussion, decisions where 
information was readily available to use and those where 
decision-making tools were more often used. 
Conclusions: The research indicated that coproduction 
may be an important factor for satisfaction with 
decision-making in the commissioning of healthcare 
services. 
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INTRODUCTION 

Commissioning, evidence and decisions in 
the National Health Service 

In England in 2013, the responsibility for 
commissioning health services changed 
hands as Clinical Commissioning Groups 
(CCGs) took over the role from Primary 
Care Trusts (PCTs). 1 It is the duty of 



Strengths and limitations of this study 



This study had a high-response rate to the 
survey which informed model development. 
Several predictors were significantly associated 
with decision satisfaction. 
A retrospective design was used which required 
participants to recall events in the past. This may 
have led to bias in responses received 
Predictors were not measured on the same scale 
as one another and so limited comparisons can 
be made of their relative effect sizes. 
The relationship between decision satisfaction 
and decision quality is not yet tested. Therefore 
we cannot conclude that a decision with which a 
commissioner is satisfied is necessarily an inde- 
pendently verifiably 'good' decision. 



National Health Service (NHS) commis- 
sioners to plan, fund and review a wide 
variety of health services ranging from emer- 
gency care to community-based interventions 
for their local populations. 2 Previous 
research on commissioning suggests that 
commissioning decisions should involve the 
drawing together of different professionals 
and interests around the 'common cause' of 
services which can better meet patients' 
needs. 3-5 Previous research also suggests that 
commissioning is complex. Decision makers 
are required to take into account a number 
of factors including local need, available 
resources, funding opportunities and need 
for savings, as well as sources of information 
such as national policy directives and avail- 
able evidence. 6 7 Elliott and Popay 8 in a pre- 
vious investigation of decision-making by 
local NHS policy makers found that the 
'influence of research evidence on decision- 
making was tampered by factors such as 
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financial constraints, shifting timescales and decision 
makers' own experiential knowledge. They suggest that 
research is 'more likely to impact on the (local) policy 
in indirect ways', including for example shaping the 
policy debate. As Walshe and Rundall 9 noted many man- 
agerial decisions in healthcare are 'constrained, con- 
tested and political'. 

Sainfort and Booske 10 reported that the process of 
measuring satisfaction with a decision is fundamental for 
difficult situations where there is no 'right' decision and 
or where long-term consequences are uncertain. This is 
the case in healthcare, particularly in commissioning 
decisions which are frequently criticised due to the sub- 
stantial variability observed across England. 11 We used 
the background literature on organisational, clinical and 
individual satisfaction with healthcare decisions and case 
study evidence to develop the concept of decision 
satisfaction. 

Often there is a lack of relevant data about existing 
populations and services and a scarcity of evidence 
about the outcomes of services which cause problems 
for those wishing to make effective decisions. These pro- 
blems, coupled with the statutory national obligations 
that need to be fulfilled, increase the pressure on newly 
formed CCGs. 

Evidence-based decision-making and coproduction 

Most research on evidence has focused on its uptake in 
health services, investigating for example the extent to 
which clinical guidelines are used. 12 13 Of equal import- 
ance is how evidence is used at a point when decisions 
have to be made and to what extent decision makers are 
satisfied with the decision outcome. 

Baumbusch et al 14 introduce the idea that the transla- 
tion and utilisation of research in clinical settings is a 
process requiring collaboration and dialogue described 
as a 'collaborative model'. Successful commissioning 
decisions are rarely made by a single individual or pro- 
fessional discipline. 7 They are the product of multiple 
views, experiences and resources. For the purposes of 
this research we use the term coproduction as defined 
by Swan et al 15 explaining this process of incorporating 
multiple views, experiences and resources into commis- 
sioning decisions. 

Proponents of a coproduction perspective discuss the 
difficulties of making academic findings useful in prac- 
tice. 16 17 They view the gap between evidence and prac- 
tice as a result of the way academics produce knowledge, 
leading to difficulties in its usefulness to practitioners. 
Hence this becomes a knowledge assimilation or transla- 
tion problem, rather than a knowledge diffusion 
problem. 18 

To overcome this issue knowledge needs to be 
coproduced by the relevant communities involved in 
commissioning. 19 20 There is no agreed definition of 
coproduction although we have identified several 
principles that are important for its success as shown in 
figure l. 15 



Embedded in 
diverse decision 
making contexts/ 
locuses 



As part of practices, 
with practitioners being 
the main actors, collective 
understandings and 
justification criteria 



As a collaboration as 
well as contested 
effort of 
co-production 
involving multiple 
parties 




Entangled in 
complex, and 
politically sensitve 
decision making 
processes 



Experienced in 
particular ways by, 
practitioners who 
need to convert 
evidence into a tool 
for everyday practice 



Figure 1 Principles important for the success of 
coproduction as defined by Swan ef a/. 15 

The aim of this research was to undertake an assess- 
ment of the association between coproduction and satis- 
faction with decisions made for local healthcare 
communities by healthcare commissioners. To achieve 
this, we developed and tested a coproduction scale and 
measured individual commissioners' satisfaction with 
commissioning decisions. 



The conceptual model 

We undertook an in-depth qualitative investigation of 
commissioners working in PCTs in England. 21 This 
empirically grounded understanding of how evidence is 
utilised in commissioning decisions enabled the develop- 
ment of a conceptual model presented in figure 2. 

The model shows six decision predictors which could 
influence satisfaction with a decision. These are 
grouped into those related to the decision, that is its 
size; the characteristics of decision makers — such as 
their background (clinical or managerial); the type of 
evidence used in the decision — practical (based on pre- 
vious experience) or empirical and the extent of 
coproduction in the decision. We hypothesised that 
these variables would be associated with decision 
satisfaction. 



METHODS 
Survey design 

This research was part of a larger study to examine the 
use of evidence for management decisions in PCTs. 15 

A cross-sectional survey of commissioners' decision- 
making was designed to test the conceptual model. 
Prepiloting and piloting of the questionnaire were con- 
ducted with purposive samples of participants drawn 
from local NHS organisations (see online supplementary 
file 1 for a copy of the survey) . The results were used to 
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Figure 2 Conceptual model of potential predictors of 
decision satisfaction. Adapted from Swan 15 (NHS, National 
Health Service). 



develop and refine the questionnaire and the process of 
administration. 

Topic areas and questions were derived from pub- 
lished surveys, literature reviews and our own in-depth 
case study evaluation of commissioning processes in four 
PCT sites. 7 15 The survey included subsections on demo- 
graphic details, work role, sources of evidence use, deci- 
sion characteristics (size, monetary value and tool 
use) 7 21 and satisfaction with decision-making using an 
adapted Decisional Conflict Scale. 22 We also drew on a 
scale measuring empirical evidence sources by 
Weatherley (see online supplementary appendix 1). 

Commissioners were asked to select a recent commis- 
sioning decision which they had been involved in and 
where the decision-making process was largely com- 
pleted. They were asked about the extent of coproduc- 
tion in the decision using specific questions presented 
in figure 3. 



We were able to share knowledge and 
Information effectively 

We were able to use the information 
effectively 

I had a sense of being involved 

There was extensive discussion 

The discussion helped us to make progress 

Many different viewpoints were explored 

People use terminology that I was not 
familiar with 

We paused discussions to clarify the meaning 
behind certain terms 

Individuals explained unfamiliar concepts and 
terms where necessary 

External information had to be significantly 
adapted to fit the problem and local context 



1 2 

1 2 

1 2 

1 2 

1 2 

1 2 

5 4 

1 2 

1 2 

1 2 



Sample 

A sample size calculation indicated that approximately 
300 respondents would be required to allow us to detect 
a 10-15% difference in proportions (with 80% power 



Figure 3 The coproduction questions and the scoring 
system applied. 



and a 95% CI) in responses by professional work role 
(ie, clinically vs non-clinically qualified commissioners). 

Participants 

We identified all potential participants in a random 
sample of 15 PCTs, stratified by the size and index of 
multiple deprivation of the population they served 
(from the total of 143 eligible PCTs excluding pilots). 
Contact details of all staff employed at NHS grade 7 
(broadly team manager or advanced practitioner level) 
or above who were involved in commissioning decision- 
making were obtained from each identified PCT. This 
included staff from departments of public health, 
finance, purchasing, commissioning, contract monitor- 
ing and information services as well as the executive 
team. 

Participants were given information sheets and details 
about how to participate. They could complete the 
survey via face-to-face meetings held at their office or by 
emails using an online electronic questionnaire. Four 
additional reminders were sent to non-respondents at 
two weekly intervals. Questionnaires completed manually 
and electronically were anonymised and transferred to 
an Excel database. 7 



ANALYSIS 

Reliability and validity of the coproduction scale 

The scale was validated and checked for subscales. To 
do this, all coproduction items were entered into a 
correlation-based principal component analysis (PCA) 
model. We hypothesised that potential subscales would 
not be independent of each other; hence PCA with 
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direct oblimin rotation was utilised to allow for low- 
factor correlation. Parallel analysis was conducted to 
identify how many components should be extracted 
from the model. Scale reliability was measured using 
Cronbach's a. PCAwas conducted using R (http://www. 
r-proj ect.org; please see online supplementary file 2 
table SI for factor loadings). 

Decision satisfaction statistical model 

Multilevel (ML) linear regression modelling was used to 
predict the decision satisfaction score using an adapted 
Decisional Conflict Scale, 15 where lower scores denote 
higher decision satisfaction. ML modelling allows us to 
model individual responses while allowing for differ- 
ences between the PCTs to also be modelled. The scores 
for the dependent variable and all predictors in both 
analyses were checked for normality using visual inspec- 
tion of histograms and Q-Q plots, alongside measure- 
ments of skewness and kurtosis. MLwiN V2.22 was used 
for the ML modelling. 

Predictors considered for inclusion in the model are 
taken from the conceptual model shown in figure 2, 
which resulted from previous qualitative research in the 
field. 21 To reduce the correlation between the predictors 
in the model, coproduction was modelled by the sub- 
scales (PCA components) rather than the individual 
items. These are detailed further in table 1. A likelihood 
ratio test was used to compare the null single-level 
model with the null ML model to determine the influ- 
ence of PCT level effects. A separate model for each pre- 
dictor was run to determine which would be included in 
the main model. Predictors found to be significantly dif- 
ferent from the null model were then considered for 
inclusion in an overall model. This model was created by 
adding these predictors stepwise in descending order of 
individual impact on decision satisfaction (determined 
by the change in — 21og likelihood in their separate 
models — representing the quantity of improvement of 
model fit). Predictors were retained in the main model 



if they improved the model fit significantly (at p=0.05 
level). 

RESULTS 
Sample 

In the first recruitment wave 6 of the 15 PCTs invited 
agreed to take part, and 9 were rerandomised by strata. 
In the second recruitment wave further 5 PCTs accepted 
resulting in a final sample of 11. The survey was circu- 
lated to 440 individuals across these 11 PCTs and 345 
(78%) responded. 7 

Participant demographics 

The median age band of the participants was 45-54 
years, and 63% of the sample were female. Forty-seven 
per cent of respondents had 5 years or less experience 
in commissioning. Thirty-one per cent (n=107) of 
respondents were qualified health or allied health pro- 
fessionals, although only 1% (n=3) was currently 
employed in a clinical setting. The largest single group 
of respondents (43%, n=149) were working in commis- 
sioning and contract roles, and the remaining were 
working in public health (33%, n=114), finance (7%, 
n=24) or other related commissioning roles (15%, 
n=52). 

Selected decisions 

When asked to select a decision to frame their 
responses, the majority (n=189, 55%) of respondents 
selected 'changing the organisation or design of a par- 
ticular service'. The second most popular was a 'major 
decision on strategic direction' (n=83, 24%) and 30 
(9%) participants selected Individual Funding Requests. 

Principal components of the coproduction scale 

The parallel analysis indicated that three principal com- 
ponents (PCs) would be sufficient and hence were 
extracted. Items were considered to be part of a PC if 
absolute item loadings were >0.45. Items 1 and 10 did 



Table 1 The effect of adding each predictor separately to the null multilevel model of decision satisfaction 



Predictor 


Improvement to model fit Coefficient B 
(change in -2log likelihood) (SE) 


Decision size (service cost) 


51.8*** 


0.004 (0.053) 


PCA1: productive discussion 


48.5*** 


-0.170 (0.023) 


PCA2: information availability and use 




-0.112 (0.023) 


The number of decision-making tools used 


1 74*** 


-0.0408 (0.0166) 


Experience of NHS commissioning (years) 


12.8*** 


-0.0102 (0.0049) 


Sources of empirical evidence as defined by Weatherly etaP 3 


10.7** 


-0.037 (0.051) 


Sources of evidence derived from our qualitative research 21 


10.4** 


-0.014 (0.050) 


Respondent medical qualification (yes/no) 


6.4* 


-0.1299 (0.0510) 


Index of multiple deprivation of population served (IMD) 


2.9 


0.0056 (0.0032) 


PCA3: dealing with uncertainty 


0.7 


-0.008 (0.024) 


Size of population served (proxy for size of commissioning organisation) 


0.1 


0.0000 (0.0000) 



*p<0.05; **p<0.01 ; ***p<0.001 . 

NHS, National Health Service; PCA, principal component analysis. 
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not load onto any component and were excluded from 
the analysis. Items were predominately well explained by 
the model (item communality > 0.5), but the two items 
(Q13 and Q17) with low communalities (less than 0.3) 
were also removed. Cronbach's a for the remaining 
items was then calculated to be 0.84, indicating very 
good reliability. The reliability for the three subscales 
was also good, with a=0.77 for PCI, 0.80 for PC2 and 
0.68 for PC3. 

The three PCs explained three distinct subscales 
centred on 'productive discussion', 'information' and 
'dealing with uncertainty', these are shown in figure 4. 
As these subscales have a maximum correlation of 0.36 
with each other, this reduces the chance of overriding 
when used as independent variables in a regression 
model. 

Decision satisfaction 

The modified decisional conflict scores were found to 
be non-normal (leptokurtic) and hence the data were 
transformed using a natural logarithm to meet the nor- 
mality assumptions of the linear model. 

The null ML model was an improvement in the null 
single-level model (change in -2LL=294.9-290.7, % 2 (1) 
=4.2, p<0.05), with a variance partition coefficient of 0.1 
indicating that 10% of the variation in decision conflict 
scores can be explained by the PCT to which the 
respondent belongs. 

The effect of adding each individual predictor to the 
model is shown in table 1. The greatest model improve- 
ment was found by adding either the decision size or 
coproduction score to the model. 

Overall, the results presented in tables 1 and 2 demon- 
strate that higher decision satisfaction was associated 
with smaller decisions, more productive discussion, deci- 
sions where information was readily available to use and 

Subscale 1: Productive discussion 

0.6: 1 had a sense of being involved 

0.7: There was extensive discussion 

Q8: The discussion helped us to make progress 

Q9: Many difference viewpoints were explored 

0.14: We were able to reach agreement 

0.15: The decision was not what I expected at the outset (negative 
correlation) 

0.16: (reversed scoring) The decision outcome was dominated by one 
group/ faction/ individual 

Subscale 2: Information availability and use 

0.2: The right people were involved 

0.3: We had sufficient information available 

Q4: We were able to share knowledge and information effectively 

0.5: We were able to use the information effectively 

Subscale 3: Dealing with Uncertainty 

Oil: We paused discussions to clarify the meaning behind certain terms 
012: Individuals explained unfamiliar concepts and terms where 
necessary 

Figure 4 The three distinct subscales explained by principal 
components (PCs) produced explained by three PCs of the 
coproduction scale. 



Table 2 The final model for influences on decision 




satisfaction (model fit -2LL=157.7) 



Predictor 


Coefficient 


(SE) 


The size of the decision 


0.021 


(0.027) 


PCA1 : productive discussion 


-0.16 


(0.02) 


PCA2: information availability 


-0.11 


(0.02) 


and use 






The number of decision-making 


-0.007 


(0.02) 


tools used 






Respondent years experience 


-0.009 


(0.005) 


of NHS commissioning 






Respondent medical qualification 


-0.09 


(0.05) 



Note that lower scores denote higher decision satisfaction. At an 
individual level, the coefficients can be interpreted as the change 
in decision satisfaction for a unit change in the predictor. 
NHS, National Health Service; PCA, principal component analysis. 



those where decision-making tools were more often 
used. Furthermore respondents with a medical qualifica- 
tion, and with great experience in NHS commissioning, 
are more likely to report greater decision satisfaction. 



DISCUSSION 

The models produced here show that several predictors 
are significantly associated with decision satisfaction. We 
found that decision satisfaction was influenced by the 
cost implications of the decision, and the scores on PCs 
of coproduction 1 and 2 (productive discussion and 
information availability and use), and the number of 
decision-making tools used. The term coproduction can 
be interpreted in a variety of ways and it is not easily 
defined as a concept. 15 20 By using the definition previ- 
ously identified by Swan et al this research indicates that 
coproduction in commissioning may consist of three 
separate components: productive discussion, informa- 
tion availability and use, and dealing with uncertainty. 

The third PC, 'dealing with uncertainty', was not 
found to influence decision satisfaction. This appears to 
indicate that uncertainty, characterised by pausing dis- 
cussions to clarify the meaning behind certain terms 
and explaining unfamiliar concepts and terms where 
necessary, was not important in decision satisfaction. It 
was not significant when included as the only predictor 
in the model indicating that the lack of effect is not 
driven by a model containing similar or overlapping 
terms. The variability of the coefficient for 'dealing with 
uncertainty' in the model of decision satisfaction was 
similar to the variability for the coefficients of 'product- 
ive discussion' and 'information availability and use'. 
Therefore the lack of effect is unlikely to be explained 
by uncertainty affecting some respondents in a positive 
way and other respondents in a negative way 

The results of our model demonstrate that increasing 
coproduction may be able to increase satisfaction with 
decisions made by healthcare commissioners. Many 
healthcare decisions are complex and difficult. The 
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'right' decision is not always obvious at the time the deci- 
sion is made. This highlights the importance of facilitat- 
ing and encouraging coproduced decision-making 
within the newly formed CCGs. This supports previous 
research that concluded that CCGs will need to make 
sure that they use "collaborative discussion and service 
planning in addition to transactional work particularly 
in times of financial austerity". 4 

Strengths and limitations of the findings 

This study had a high response rate to the survey which 
informed the model development. Nevertheless, the 
research used a retrospective design which required par- 
ticipants to recall events which had happened in the 
past. There may have been bias in responses received 
due to recall bias. Social desirability bias may also have 
been introduced as participants may have given answers 
which they deemed to be appropriate to the researchers 
— not necessarily which reflected their true beliefs about 
the situation. Given the nature of the data collection 
and our promises of anonymity, it was impossible to cor- 
relate reports by different managers on the same deci- 
sion, although this would have lent additional validation 
to our findings. Also, we do not know the effect on the 
results if patients' and providers' satisfactions with the 
decisions had been surveyed. 

Although the models show that several predictors are 
significantly associated with decision satisfaction, these 
predictors (with the exception of the three parts of 
coproduction) are not measured on the same scale as 
one another which means that limited comparisons can 
be made of their relative effect sizes. It is also possible 
that the elements identified might reflect an underlying 
latent factor, or that other important factors such as 
good working relationships are involved. These issues 
will be of interest for further research in this area. 

Implications for practice and future research 

The findings of this research are important for commis- 
sioners in the NHS today who operate in a difficult and 
changing environment. Commissioners must acknow- 
ledge the implications that their decisions have on the 
health of the population for which care is being 
commissioned. Those working in, or managing commis- 
sioning organisations need to: 

► Include the right individuals in the decision-making 
process; 

► Ensure access to the right data and resources; 

► Ensure that decision-making tools are available to 
commissioning groups. (Tools currently available 
include clinical guidelines, cost-effectiveness analyses 
and National Service Frameworks. 7 ) 

► Facilitate productive discussion and exploration of dif- 
ferent views throughout the decision-making process. 

Ensuring that a decision is coproduced is a step in the 
right direction towards bridging the research to practice 
gap. Collaborative decision-making using the coproduc- 
tive approach enables knowledge to be moved across 



boundaries between academic community and society, 
taking into account the interests and objectives of a 
range of stakeholders. 13 24-26 

Further research should validate and investigate mea- 
surements of decision satisfaction in this organisational 
setting and in other similar settings and in this context 
it would be of interest to investigate the relationship 
between commissioners' patients' and providers' satisfac- 
tion with the same decisions. 

CONCLUSION 

Organisational decision-making to plan services is import- 
ant in all healthcare systems but is often difficult in prac- 
tice. Satisfaction with commissioning decisions in this 
research required coproduction in the form of collabor- 
ation, interaction and using the 'right' resources. Our data 
demonstrate that coproduction is comprised of three sep- 
arate components (productive discussion, information 
availability and dealing with uncertainty) . In this research, 
productive discussion appeared more important than the 
information availability and use for decision satisfaction. 
These findings will be of value to CCGs in commissioning 
decision-making and the use of evidence, as they make 
decisions for the benefit of their local populations. 
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